Agentic perf skills: polish + retire select-agent-models and agentic-perf-reviewer agent - #838

Read-only audit skill for .NET agentic apps. Runs seven check classes

(topology, tool inventory, message-history strategy, prompt weight,

parallelism, OTel coverage, model assignment) and writes a severity-tagged

Markdown report to .copilot/perf-reports/audit-<timestamp>.md plus

latest-audit.md.

Findings carry file:line citations, evidence, a one-paragraph why,

and a concrete next action that can route into select-agent-models,

setup-maf-evals, or configure-agentic-perf-rules. Never edits source.

Includes SKILL.md, eight reference docs, eval.yaml with five scenarios

(clean / sprawl / full-history / prompt-bloat / missing-otel),

and a CODEOWNERS entry matching the dotnet-ai convention.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Recommends per-agent model assignments based on classified roles

(router, planner, decomposer, worker, validator, formatter, summarizer)

and a curated role->model matrix balancing latency, quality, and cost.

Two modes: read-only recommend (default, writes

.copilot/perf-reports/model-plan-<timestamp>.md) and apply

(diff-preview-and-confirm). Never applies without explicit confirmation.

Includes SKILL.md, role-model matrix, AppHost multi-client + per-agent

resolution templates, plan template, and eval.yaml with four scenarios

(recommend mode / single-model defaulting / single-agent abort /

apply mode requires confirmation).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Scaffolds an <App>.Evals project alongside an existing .NET agentic app

(MAF + Aspire + Foundry) with three modes: telemetry (per-call latency /

tokens / cost), quality (LLM-as-judge against a rubric and golden

conversations), and compare (two model assignments side by side).

Outputs Markdown + JSON + JUnit-XML to .copilot/perf-reports/evals/.

Optionally adds an Aspire-served static-file panel showing per-agent

token/latency live during dev (apply mode required for AppHost edits).

Includes SKILL.md, project template, telemetry capture, quality modes,

compare mode, optional dashboard panel, and eval.yaml with four scenarios

(scaffold fresh / skip non-agentic / stub run / update preserves user edits).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Companion architect agent for .NET agentic apps (MAF + Aspire +

Foundry). Runs a three-pass review:

Pass 1 — direct read (no skills): inventory, topology, obvious smells.

Pass 2 — deep audit: orchestrates audit-agentic-app-perf, then routes

into select-agent-models / setup-maf-evals / configure-agentic-perf-rules

based on the report.

Pass 3 — synthesis: prioritized action list with skill routes,

expected effects, and validation steps.

Registers the agent path in plugins/dotnet-ai/plugin.json (and the

.codex-plugin manifest) using an explicit file path per the validator.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Addresses critique findings:

- project-template.md: remove invalid semicolons in <None Update>

  attribute lists (was breaking MSBuild parse), add explicit Version

  attributes to PackageReferences with CPM fallback note, add

  ProjectReference to agent-under-test, define IEvalRunner +

  EvalReport contract that Program.cs depends on.

- SKILL.md: split step 1 into discover + update-mode matrix that

  explicitly classifies infra vs user-data vs generated files.

- SKILL.md: rewrite step 7 (Aspire panel) as an explicit apply-mode

  step with diff-preview-and-confirm; declined path scaffolds files

  outside the AppHost.

- telemetry-capture.md: split EVAL_USE_REAL_MODELS into two toggles

  EVAL_USE_REAL_AGENT and EVAL_USE_REAL_JUDGE so quality/compare

  modes have a coherent stub story.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Addresses critique findings:

- SKILL.md step 4: drop the .gitignore edit instruction. The skill

  now only writes under .copilot/perf-reports/. Recommendation about

  ignoring the report folder is surfaced in chat only.

- SKILL.md step 3: add explicit evidence gate that requires

  re-reading the cited file before emitting a finding. Adds

  check_id (T1/T2/...) to the schema for deterministic referencing

  across reports and reviews.

- SKILL.md pitfalls: spell out which files are off-limits explicitly

  (.gitignore, source, config) and require evidence verification.

- references/report-template.md: render check_id in finding heading.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Addresses critique findings:

- Rename heading from Two-Pass to Three-Pass to match the actual

  number of passes defined.

- Remove edit from the tool list. Agent claims no source edits in

  its boundaries; the tool is the actual mechanism by which it

  could violate that boundary, so removing it eliminates the

  footgun.

- Add explicit exclusion in the description so routing prefers

  optimizing-dotnet-performance for non-agentic .NET perf reviews.

- Pass 1 step 1: define detection signals concretely (AppHost,

  Microsoft.Agents.AI / Microsoft.Extensions.AI references,

  ChatClientAgent, Foundry config) so Pass 1 can stop cleanly when

  the project is not agentic.

- Pass 2 step 2: replace MA-/OTel-category language with the

  actual report schema (check_id MA1-MA4, O1-O4) and require the

  agent to read the report file rather than infer from memory.

- Pass 3 step 3: require qualitative effects unless setup-maf-evals

  has produced a citable report; no fabricated numeric estimates.

- Boundaries: clarify apply-mode chaining — user intent in initial

  turn is not user confirmation; the invoked skill still owns its

  own confirmation flow.

- Add explicit quick-triage escape hatch so Pass 2 isn't unconditional

  when the user has asked for a fast read.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Addresses critique findings:

- SKILL.md frontmatter: add explicit 'version: 0.1.0' field as the

  authoritative source. Update SKILL.md step 2 references to read

  the version from frontmatter rather than assume.

- SKILL.md step 2: spell out sentinel parse rules with exact

  full-line regexes for BEGIN and END, refuse-on-malformed contract

  (zero/multiple/mismatched/out-of-order sentinels), and a

  same-version-but-structurally-invalid path that no longer

  silently no-ops when the block is missing rule headings or has

  unparseable threshold YAML.

- SKILL.md step 2: replace the brief 'merge thresholds' phrase with

  a concrete five-step preservation algorithm (parse, validate

  types, override known keys, drop unknown with warning, write).

- SKILL.md step 2: add path-safety rules (resolve project root,

  reject paths that normalize outside it, reject absolute paths

  and .. segments).

- SKILL.md: add target-file selection table for the

  AGENTS.md vs .github/copilot-instructions.md case so 'both have

  a managed block' refuses cleanly instead of doubling up.

- SKILL.md frontmatter: relax 'with Aspire and Foundry' to

  'optionally with Aspire/Foundry' so MAF-only projects route

  correctly (matches the eval fixture).

- references/managed-block-template.md: switch outer fence from

  three to four backticks so the inner three-backtick yaml fence

  in the rendered template is no longer ambiguous to Markdown

  parsers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Addresses critique findings:

- SKILL.md step 6: split apply mode into seven explicit sub-steps.

- Add provider-resolution step that distinguishes public OpenAI from

  Azure OpenAI / Foundry deployments by inspecting AppHost connection

  string and appsettings keys, and refuses to write a public OpenAI

  id into an Azure project's appsettings.json. The agent must ask the

  user which deployment alias maps to each recommended role or stop

  apply mode if no suitable alias exists.

- Add explicit confirmation contract that distinguishes the initial

  apply request (intent) from the confirmation that follows the diff

  preview, so phrasings like 'apply and confirm' or 'just do it' do

  not collapse the two-step contract.

- Add pre-write parse / build-dryrun validation step before any write.

- Add atomic-write contract: if any write fails midway, restore all

  touched files from their pre-write content.

- Add build-failure rollback: on dotnet build failure after apply,

  revert all written changes and report apply failed (build) instead

  of declaring success.

- references/role-model-matrix.md: add router sub-types section that

  promotes the router from gpt-4o-mini to gpt-4o / o4-mini when the

  router generates tool args, validates input schema, has more than

  five destinations, or routes to expensive workflows.

- references/role-model-matrix.md: add planner decision criteria for

  o4-mini vs gpt-4o based on dependency depth, latency budget, and

  output structure.

- references/role-model-matrix.md: add multi-role classification

  table for agents that fit more than one role.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Dogfood pass against interview-coach-v2 / behavioral-interview-coach

surfaced that the project-template.md illustrative versions

(9.4.0-preview.1.*, Microsoft.Extensions.* 9.0.0) were significantly

stale. Verified via 'dotnet package search Microsoft.Extensions.AI.Evaluation'

that the current GA family is 10.7.0 across:

- Microsoft.Extensions.AI

- Microsoft.Extensions.AI.Evaluation

- Microsoft.Extensions.AI.Evaluation.Quality

- Microsoft.Extensions.AI.Evaluation.Reporting

Switch the template to 10.* floating versions and clarify in the

lead-in that the listed numbers reflect the latest stable family

(not just an arbitrary placeholder). The skill body still requires

the agent to query nuget.org at scaffold time.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

User feedback: the separate category field was confusing because

the check_id prefix already encodes the category (T=topology,

MH=history, etc.). Also: the skill produced a report and stopped

without offering to route into the apply-capable downstream skills.

Changes:

- SKILL.md step 3: remove category field from finding schema. The

  check_id prefix is now the sole category indicator. Add a glossary

  table mapping each prefix to its category and reference doc.

- SKILL.md step 4: sort findings by check_id within severity bands

  instead of by category.

- SKILL.md step 5: render findings with check_id + title (no

  parenthetical category).

- SKILL.md add step 6 (Offer to route): after surfacing top findings,

  ask the user once whether they want to follow up on any routed

  ref: targets. Render only the lettered options that correspond to

  refs actually present in the findings. Explicitly does not edit

  source — just routes into the appropriate sibling skill which

  owns its own diff-and-confirm flow. Apply-mode chaining language

  mirrors the agent: intent is not confirmation.

- SKILL.md step 7: previous step 6 (Stop) renumbered.

- references/report-template.md: render finding heading as

  '[severity] [check_id] title' and add a glossary line under the

  ## Findings heading. Update the per-finding block to drop the

  separate Check field.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

… follow-ups in Pass 3

Pairs with the audit-agentic-app-perf change that drops the separate

category field. The agent's Pass 2 routing rules now read the check_id

prefix (MA*, O*, etc.) instead of looking for a 'category' column

that no longer exists in the report.

Pass 3 also gains a final sub-step that asks the user once whether to

run any of the routed skills now, mirroring the new step 6 in the

audit skill. Apply-mode chaining language is preserved: the invoked

skill still owns its own diff-and-confirm.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

User feedback: 'audit' overlaps with security audits and the

companion agentic-perf-reviewer agent's review framing. 'Scan' is

shorter, conveys the read-only sweep more clearly, and avoids the

overlap. Internal narrative still uses 'audit' as a synonym where

natural; trigger phrases keep both.

- Rename plugins/dotnet-ai/skills/audit-agentic-app-perf -> scan-agentic-app-perf

- Rename tests/dotnet-ai/audit-agentic-app-perf -> scan-agentic-app-perf

- SKILL.md: name field, # heading, lead description verb (Scan...),

  trigger phrases (added 'scan my agentic app', kept 'audit my...

  agentic app' for back-compat search)

- Report filenames: audit-<timestamp>.md -> scan-<timestamp>.md,

  latest-audit.md -> latest-scan.md (changes the on-disk artifact

  the agent expects to read; agent.md updated on its branch)

- references/report-template.md: filename refs

- eval.yaml: skill key + prompts

- CODEOWNERS: path entries

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Pairs with the rename on skills/audit-agentic-app-perf branch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Pairs with the rename on skills/audit-agentic-app-perf branch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Pairs with the rename on skills/audit-agentic-app-perf branch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Pairs with the rename on skills/audit-agentic-app-perf branch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…detection)

Rewrites SKILL.md and references to specify the v2 overhaul:

- Replaces hand-rolled markdown runner with a Microsoft.Extensions.AI.Evaluation.Reporting + `aieval` HTML report pipeline (GA 10.7.0).

- Switches the scaffolded project from a console runner to an MSTest `<App>.Evals.Tests` project matching the canonical Learn-docs pattern.

- Categorizes evaluators into three independent tiers (NLP / Quality / Safety) with separate env knobs (EVAL_USE_REAL_AGENT, EVAL_USE_REAL_JUDGE, EVAL_USE_FOUNDRY_SAFETY).

- Adds IChatClient auto-detection so generated AgentChatClientFactory wires to the app's existing chat client registration.

- Adds opt-in Safety tier (ContentHarmEvaluator via Azure AI Foundry) and opt-in GitHub Actions workflow.

- New eval.yaml has 8 scenarios covering scaffold, update-mode preservation, IChatClient detection, reporting wiring, schema v2, CI workflow, safety opt-in.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Three gaps surfaced when scaffolding ELI5Agent.Evals.Tests against an Aspire 13.2 + Foundry app:

1. **Detection table missing the Aspire Inference pattern.** `builder.AddAzureChatCompletionsClient(\"chat\").AddChatClient(\"chat\")` is the standard Aspire 13.2 way of wiring an IChatClient against a Foundry chat deployment, and was not in the v2 detection table. Added to ichatclient-detection.md with a note that the argument is the connection-string name.

2. **Connection-string setup not surfaced for standalone runs.** When the app uses Aspire orchestration, `ConnectionStrings:<alias>` is populated by the AppHost — but `dotnet test` runs outside the host and gets a silent missing-config NRE on first real-agent run. Added a `Connection-string setup for standalone test runs` section to ichatclient-detection.md that surfaces both user-secrets and env-var setup paths and points to `azd env get-values`.

3. **WordCountEvaluator implementation not pinned.** The catalog called the evaluator out as `always scaffolded (custom)` but never gave a verbatim template. Added the Learn-doc canonical implementation to evaluators-catalog.md so every scaffold gets the same (correct) IEvaluator skeleton.

Validated end-to-end against ELI5Agent: scaffolded ELI5Agent.Evals.Tests, `dotnet test` exits 0 in stub tier with 4 scenarios x 4 metrics (Words/BLEU/GLEU/F1) producing a 670 KB report.html. Real-judge tier deferred (no Azure creds set up locally for ELI5Agent — itself the trigger for finding 2).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…gnostic

1. New references/metrics-glossary.md authored as the source of truth for metric definitions/scales/thresholds across NLP, Quality, and Safety tiers. Includes the canonical Reporting/MetricsGlossary.cs template that the skill emits into <App>.Evals.Tests/Reporting/.

2. Factory template in references/ichatclient-detection.md now wraps DI resolution in try/catch and throws a friendly InvalidOperationException naming the connection-string key + the exact 'dotnet user-secrets set' command + the env-var alternative + 'azd env get-values' pointer. Replaces the silent NRE that ELI5Agent dogfooding hit when EVAL_USE_REAL_AGENT=1 with no creds set up.

3. Two new eval.yaml scenarios bind the new behavior:

   - scaffold-emits-metrics-glossary: asserts MetricsGlossary.cs exists, references metrics-glossary.md, has [AssemblyCleanup]

   - factory-emits-friendly-secrets-diagnostic: asserts AgentChatClientFactory mentions 'dotnet user-secrets' + 'ConnectionStrings'

4. Captured two MSTest constraints that turned up while implementing the glossary writer:

   - quality-modes.md ReportingConfig: ExecutionName must be cached at class load, NOT re-evaluated per call (otherwise AievalReport and MetricsGlossary land in different timestamped folders 5 s apart)

   - metrics-glossary.md template now declares MetricsGlossary as a plain static class; MSTest forbids more than one [AssemblyCleanup] per assembly (UTA014). Glossary write is chained from AievalReport's single AssemblyCleanup, wrapped in try/catch so a glossary-write failure doesn't mask the report.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Spec was creeping toward the validator's 'comprehensive' threshold (3,574 BPE tokens / 14,238 chars / 265 lines). Pulled prose-heavy sections into references and kept SKILL.md focused on decision-relevant content.

- Step 3 (Scaffold): replaced 27-line file tree with a one-sentence summary + link to project-template.md. Kept the post-write powershell as a fenced code block so the agent has concrete commands.

- Steps 4-9 (telemetry/quality/compare/safety/panel/CI): collapsed bulleted prose into 3-4 line stubs that retain the decision facts (default ON/OFF, opt-in semantics, env knob) and link the corresponding reference.

- Step 11 (Surface in chat): added the metrics-glossary.md path to the 'Paths' bullet and to the trailing 'see also' line.

- ## Common pitfalls: extracted to references/common-pitfalls.md (also adds two new entries from this work: the multi-AssemblyCleanup MSTest constraint and the AgentChatClientFactory friendly-NRE pattern). SKILL.md keeps a one-line link.

- ## References: trimmed each bullet to one short line; consolidated the three external links onto a single line.

Net effect: SKILL.md 14,238 -> 10,449 chars, 265 -> 189 lines, 3,574 -> 2,672 BPE tokens (-25%). Validator still passes; eval.yaml scenarios are unaffected (assertions are file-based, not prose-based).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Dogfood findings against ELI5Agent surfaced 3 real failure modes when
promoting from stub to judge tier:

1. user-secrets silently not loading: dotnet test runs under testhost.exe
   as the entry assembly, so Host.CreateApplicationBuilder() does NOT
   pick up the secrets store keyed off the test project's UserSecretsId.
   Fix: factory template now calls
   builder.Configuration.AddUserSecrets(typeof(...).Assembly, optional: true).

2. services.ai.azure.com hostname strips dashes from the resource name
   (foundry-abc -> foundryabc.services.ai.azure.com). The legacy
   properties.endpoint value points at cognitiveservices.azure.com which
   404s for the /models route.

3. Foundry resources provisioned by Aspire/azd usually have
   disableLocalAuth=true. Key-based auth returns 403; drop the Key=
   segment and rely on DefaultAzureCredential.

Updates:
- references/ichatclient-detection.md: factory template adds AddUserSecrets;
  diagnostic message now lists Entra and Key options + endpoint gotchas.
- references/project-template.md: GlobalUsings.cs adds Microsoft.Extensions
  .Configuration / .DependencyInjection / .Hosting (needed by the factory).
- references/common-pitfalls.md: 3 new entries covering the above.
- SKILL.md: step 11 now surfaces the exact 2-command judge-promotion path.
- tests/dotnet-ai/setup-maf-evals/eval.yaml: new
  factory-loads-user-secrets-explicitly scenario.

Verified end-to-end: ELI5Agent judge tier 4/4 passing in 2m 24s,
report.html shows 9 metric families (Words/BLEU/GLEU/F1 + Relevance/
Coherence/Fluency/Completeness/Equivalence).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Dogfooding the Judge tier against ELI5Agent's gpt-5-mini judge surfaced
a silent failure: reasoning models (gpt-5*, o-series) reject the
max_tokens parameter that Azure.AI.Inference still sends, returning
HTTP 400 unsupported_parameter. The MEAI Quality evaluators swallow
the 400 and record it as a per-metric error row -- tests exit 0 but
every Quality column is an error.

Pitfalls doc now covers:
- which model families are affected (gpt-5*, o1/o3/o-series)
- how the failure manifests (test pass + errors in report.html, not
  a hard test failure)
- the workaround (pick a non-reasoning judge: gpt-4o / gpt-4o-mini /
  gpt-4-turbo; if the agent uses reasoning, set
  EVAL_JUDGE_DEPLOYMENT_NAME=<non-reasoning-alias> to split agent and
  judge deployments)
- az CLI snippet to list deployment model families

SKILL.md step 11's judge-promotion block now flags this alongside the
existing dash/key-auth gotchas. eval.yaml gets a smoke assertion that
common-pitfalls.md mentions both max_tokens and the env-var workaround.

Verified end-to-end: re-pointed ELI5Agent at a gpt-4o-mini Foundry
deployment, cleared _store/, re-ran judge tier: 4/4 passing in 44s,
clean report (no max_tokens / unsupported_parameter strings anywhere),
all 9 metric families populate (4 NLP + Relevance / Coherence /
Fluency / Completeness / Equivalence).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…n (CS9006)

Found by dogfooding the v2 template against ELI5Agent: the prompt block
used single-dollar interpolated raw string with {{ }} attempting to
escape literal JSON braces. C# raw strings reject {{ as literal in
single-dollar mode (CS9006: 'does not start with enough $ characters
to allow this many consecutive opening braces as content').

Fix: switch to double-dollar raw string $$"""...""" where literal { is
fine and {{var}} is interpolation. Added a comment in the template
explaining the choice so users don't "simplify" it back.

End-to-end validated against ELI5Agent:
- Compiles clean.
- 4/4 judge-tier tests pass in 58s with the new evaluator wired.
- RubricFit column appears in report.html with substantive rationales.
- Score comparison (same agent, same responses):
    generic Relevance 3.25, Coherence 3.00, Fluency 3.50
    vs RubricFit (ELI5-tuned rubric) 5.00 / 5
  -- exactly the remediation outcome documented in common-pitfalls.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Document that 'Cache Miss' on every call is expected in the
real-agent + real-judge workflow because the judge cache key
includes the agent's response, which varies run-to-run. Also
note that executionName scopes the cache.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Three changes that make the MEAI response cache actually pay off
across runs (verified on ELI5Agent: cold 39s/0 hits -> warm 15s/20
hits with the same scenarios):

1. QualityTests uses run.ChatConfiguration!.ChatClient (the cached
   wrapper) for the agent call by default, only falling back to the
   uncached factory when EVAL_JUDGE_DEPLOYMENT_NAME splits judge
   from agent. Previously every run created a fresh uncached agent
   client and the agent's varying output guaranteed judge cache
   misses too.

2. Drop the executionName argument from
   DiskBasedReportingConfiguration.Create. executionName is part of
   the cache scope -- passing a fresh per-run timestamp guaranteed
   misses regardless of input. Report folder timestamping moves to
   the new EvalEnv.ReportFolder (EVAL_REPORT_FOLDER env var).

3. SKILL.md step 11 + common-pitfalls.md rewritten to explain the
   cache payoff: first run ~60s populates _store/cache/, every
   subsequent run against unchanged inputs ~5s with zero LLM cost.

Compare mode keeps a stable per-entry executionName for cache reuse
across compare runs. ichatclient-detection.md documents the
override-splits-cache trade-off.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…skills

Polish pass after dogfooding 3 perf skills against ELI5Agent (1-agent
MAF/Aspire/Foundry app):

- scan-agentic-app-perf: new check-id-glossary.md (24 check codes,
  severity, per-category file links, cross-skill route table) written
  to .copilot/perf-reports/ alongside scan reports. Mirrors the
  metrics-glossary.md pattern from setup-maf-evals v2.
- scan / configure / select: new references/common-pitfalls.md
  capturing dogfooded false-positive patterns (MA4 in AppHost is
  expected, MA1 false-fires on 1-agent apps, O3 in MEAI source is
  normal, sentinel parsing must fail-closed, role classification
  edge cases, etc).
- agentic-perf-reviewer.agent.md: inline Common pitfalls section
  (agents have no references/ folder by convention).
- tests/dotnet-ai/agentic-perf-reviewer/eval.yaml: 11 routing-
  discrimination scenarios (4 should-invoke, 4 should-defer-to-
  optimizing-dotnet-performance, 3 should-route-to-child-skill).

Validator: ✅ 9 skills + 1 agent + 1 plugin pass.
Markdownlint: ✅ 0 errors on changed files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…rences

Wrap <agent>, <tool-name>, <A>, <B>, <leaf>, <downstream>, <names>,
<tool-A>, <tool-B>, <tool>, <N> placeholder identifiers in backticks
across the 5 *-checks.md files. These had been parsed as raw inline
HTML elements by markdownlint (MD033/no-inline-html), producing 17
pre-existing errors. The intent was always literal placeholders in
the 'Next:' action templates, so backticks are the correct fix.

Verified: markdownlint-cli2 on all 5 changed files now reports 0
errors. Skill validator still passes (9 skills + 1 agent + 1 plugin).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Surfaced as a skill-gap during the code-review-buddy dogfood
(leslierichardson95/code-review-buddy Phase 2.5): users designing a
new agentic app had no skill-driven way to consult the role-model
matrix before writing any code — they had to read
references/role-model-matrix.md directly.

Plan mode is source-free:
- Triggers on 'design my topology', 'planning a new app',
  'what models should I use, haven't written code yet'.
- Asks for { agents, quality_priority, provider_constraint }.
- Single-agent plans are valid (recommend mode's <2 abort is bypassed).
- Output adds a deployment-shape section (alias -> model id, who uses it)
  + a verify-checklist driving a later recommend-mode pass for
  prediction-held confirmation.

Files written to a separate model-plan-design-<ts>.md series so the
later recommend run can sit next to it.

Skill description / WHEN / NOT-WHEN updated. references/plan-template.md
documents the new layout + relaxed empty-plan contract.

Validated against the existing dogfood:
code-review-buddy/.copilot/perf-reports/model-plan-design.md was hand-
written following exactly this layout in Phase 2.5; verify-mode
(model-plan-verify.md) confirmed every prediction held.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Resets scope after dogfood feedback. Original motivation for
select-agent-models was 'Copilot always defaults to gpt-4o when
scaffolding agents.' The skill grew well past that — recommend/plan/
apply modes, role-model matrix doc, registry-currency questions,
greenfield plan-mode follow-on, etc. — and Foundry's model-router
covers the per-request piece better than the skill ever could.

Net change: minimum viable fix lives where it always should have —
in rule dotnet#3 of the configure-agentic-perf-rules managed block that
Copilot already reads on every turn.

configure-agentic-perf-rules (bumped to v0.2.0):
- Rule dotnet#3 in managed-block-template.md is now self-contained and
  prescriptive: a role-pick table covering router/validator/formatter/
  worker -> small-fast; planner -> reasoning-class; creative ->
  frontier. Calls out Foundry model-router as the recommended worker
  pick when prompt length varies. Says 'stop and ask if unsure;
  do not silently default to gpt-4o.'
- rule-rationales.md rule dotnet#3 mirrors the new table with why-it-matters
  + when-frontier-is-justified + 'state the why in code' guidance.
- Removed all references to select-agent-models from NOT-WHEN /
  companion-skills sections.

scan-agentic-app-perf:
- MA1/MA2/MA3 'Next' actions now point at rule dotnet#3 of the managed
  block instead of select-agent-models. 'Ref:' fields updated.
- check-id-glossary.md, common-pitfalls.md, report-template.md, SKILL.md
  description + follow-up offer updated to drop the skill reference.

setup-maf-evals:
- Description NOT-WHEN no longer mentions select-agent-models.
- Follow-up recommendation after telemetry/quality runs now references
  'a model swap per rule dotnet#3' instead of the deleted skill.
- aspire-dashboard-panel.md drops the cross-reference.

agentic-perf-reviewer.agent.md:
- Description + Skills-used drop select-agent-models.
- Pass 2 model-finding route now points at configure-agentic-perf-rules.
- Pass 3 follow-up letters condensed from 4 to 3 (A=rules, B=evals,
  C=stop).
- Single-agent-app pitfall reworded: rule dotnet#3 covers single agents too,
  no abort-to-skip needed.

DELETED: plugins/dotnet-ai/skills/select-agent-models/ (6 files).

The audit case (scan-agentic-app-perf MA1-MA4) still covers existing
apps. The greenfield case is the rule itself, sitting in copilot-
instructions.md where Copilot reads it on every scaffold turn —
which is the actual problem we set out to solve.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Same simplification rationale as the select-agent-models retirement.

After moving role-aware model selection into rule dotnet#3 of
configure-agentic-perf-rules, the agent's orchestration logic
collapsed to:
  - run scan-agentic-app-perf (Pass 2)
  - route MA* findings to configure-agentic-perf-rules
  - route O* findings to setup-maf-evals

That's logic Copilot already does organically from the skills'
descriptions and the scan's existing 'ref:' fields + step-6 routing
offer. The agent was adding ~70% ceremony for ~30% real value.

Net change:
- DELETED agents/agentic-perf-reviewer.agent.md (~180 lines)
- DELETED agents/ directory (now empty)
- Removed 'agents' key from plugin.json and .codex-plugin/plugin.json
- The one unique guard worth keeping ('don't recommend a model
  downgrade without an evals quality follow-up') migrated into
  scan-agentic-app-perf SKILL.md common pitfalls.
- The other 'don't pre-confirm apply-mode writes on user's behalf'
  guard already lives in scan-agentic-app-perf step 6.4.

Users now invoke the skills directly:
  - 'scan my agentic app' -> scan-agentic-app-perf (which offers
    its own A/B/C follow-up menu)
  - 'install perf rules' -> configure-agentic-perf-rules
  - 'wire up evals' -> setup-maf-evals

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…slugs; prune arbitrary thresholds

Two related cleanups based on dogfood UX feedback.

1. Cryptic codes -> human-readable slugs.
   Findings used to be tagged with MA1, TI4, PW3, etc. — codes that
   were stable + greppable but required readers to flip to a glossary
   file every time. The prefix-routing case (e.g. 'MA* -> select-
   agent-models') that justified the convention is gone now that
   select-agent-models and the orchestrator agent are retired.

   Schema:
     check_id: T1 | T2 | TI1 | ... | MA4
   becomes:
     check: <category>.<descriptor>
   where category is one of topology / tools / history / prompt /
   parallel / otel / model and the descriptor names the actual issue.

   Examples:
     MA1 -> model.same-default
     MH1 -> history.full-share
     TI4 -> tools.description-too-long
     O1  -> otel.missing-sdk

   No glossary lookup needed — the slug self-describes.

   check-id-glossary.md -> check-glossary.md with a new
   slug -> description table.

2. Pruned arbitrary-threshold checks.
   Five checks were taste calls dressed up as audit findings; they
   fired on legitimate designs as often as on real bloat:

     - T1 'agent count > 3' (DELETED)
     - T2 'LLM handoff edges per turn > 2' (DELETED)
     - TI1 'tools per agent > 8' (DELETED)
     - PW2 'few-shot examples > 3' (DELETED)
     - P1 'sequential awaits over independent inputs' (DELETED;
       general .NET concurrency anti-pattern, belongs to
       optimizing-dotnet-performance)

   Each deprecated check leaves a 'What used to live here' breadcrumb
   in its category file so users grep-finding the old name see why
   it was removed and where the underlying concern is now addressed.

Net: 24 checks -> 19 checks, no schema gymnastics, no glossary
round-trips for the reader.

Files touched:
- references/{topology,tool-inventory,message-history,prompt-weight,parallelism,otel-coverage,model-assignment}-checks.md rewritten
- references/check-id-glossary.md -> check-glossary.md (rename + rewrite)
- references/report-template.md schema update
- references/common-pitfalls.md per-check sharpening updated for new slugs
- SKILL.md finding schema + sort key + references list updated

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ten scan.md

Dogfood feedback: three generated files per run (scan-<ts>.md +
latest-scan.md + check-glossary.md) plus a new timestamped file
every scan made even a clean repo's .copilot/perf-reports/ look like
a hoarder's basement after a week. The slug rename made the glossary
file redundant anyway — 'history.full-share' is self-describing in a
way that 'MH1' never was.

Changes:
- Single output file: .copilot/perf-reports/scan.md, overwritten on
  every run. No timestamped copies, no latest-* mirrors, no per-run
  glossary file.
- Category legend is inlined into the report (the blockquote at the
  top of '## Findings') so readers see what topology/history/model/
  etc. mean without leaving the file.
- references/check-glossary.md remains as a dev-facing catalog (for
  skill maintainers adding new checks); it is explicitly NOT copied
  into user repos.
- references/report-template.md updated with the new filename + the
  inlined legend block.
- Validation contract simplified: one file exists, has a '## Findings'
  section, counts match chat.
- Multi-AppHost case: per-host filename is now scan-<host>.md
  (still overwritten each run, no timestamp suffix).

Git is the history mechanism if anyone wants one — the skill no
longer reinvents version control by hand.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…m rules #1/#2; bump to 0.3.0

Rules #1 (agent count) and #2 (handoff edges) shipped with numeric
ceilings (agent_count_max: 3, llm_routed_edges_max_per_turn: 2) that
tripped legitimate designs (specialist-handoff workflows, multi-hop
deterministic chains with one LLM-routed intent decision) as often as
they caught real bloat. The justify-the-add gates were always the
actual mechanism; the numbers were illusory precision.

Mirrors the same prune applied to scan-agentic-app-perf (a38f3f8) where
the topology.agent-count and topology.handoff-fanout checks were removed
for the same reason.

Changes:
- managed-block-template.md: drop both keys from the thresholds YAML block;
  reword rules #1/#2 to explicitly say 'no hard ceiling — justify the add'.
- rule-rationales.md: drop the 'when you must add a 4th+' subsection;
  add a 'on thresholds' note explaining why the numeric ceilings were
  removed (so consumers reading rationales understand the v0.3.0 change).
- threshold-defaults.md: drop the first 2 rows of the threshold table;
  add a deprecation note pointing to the rules.
- SKILL.md: bump version 0.2.0 -> 0.3.0; update the threshold-preservation
  example from agent_count_max to per_turn_input_token_warn.
- common-pitfalls.md: update strict-parse example from agent_count_max
  to per_turn_input_token_warn.
- eval.yaml: drop agent_count_max literal assertion from fresh-install
  test; bump idempotency-test fixture from v0.1.0 -> v0.3.0; bump
  threshold-preservation fixture from v0.0.1 to drop deprecated keys
  from the input YAML; add new scenario validating v0.2.0 -> v0.3.0
  upgrade drops deprecated keys with a chat warning.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…l/agent entries

Replace @mikekistler with @cathysull as the second codeowner on the three
remaining perf-skills areas:
- configure-agentic-perf-rules (plugin + tests)
- scan-agentic-app-perf (plugin + tests)
- setup-maf-evals (plugin + tests)

Remove dangling CODEOWNERS entries for the retired select-agent-models
skill and the agentic-perf-reviewer agent (the corresponding files were
removed in 76cc8a5 and ecce98a but the ownership lines were missed).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Caught while dogfooding the setup-maf-evals scaffold against the
code-review-buddy reference app. The scaffolded `InputsLoader`,
`MatrixLoader`, `PriceTable.Load` and `GoldenLoader` helpers were
landing without an explicit `JsonSerializerOptions`, so PascalCase C#
records bound to snake_case JSON keys came back null. Quality mode
failed loudly; telemetry mode failed *silently* (zeroed records whose
null fields are never read), corrupting cost rollups.

Add canonical loader sketches with
`PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower` +
`PropertyNameCaseInsensitive = true` to:
- `references/telemetry-capture.md` (after the `inputs.json` schema)
- `references/compare-mode.md` (after the `matrix.json` schema)

Add a matching common-pitfalls entry pointing at both.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…I/v1 detection pattern

The Azure.AI.Inference beta SDK (the foundation for the Aspire
`AddAzureChatCompletionsClient` registration) is being retired on
August 26, 2026 per the Foundry supported-languages doc. New apps
should prefer the OpenAI/v1 API + stable OpenAI SDK via
`AddOpenAIClient(...).AddChatClient(...)`.

This commit teaches the setup-maf-evals skill about both paths:

- `references/ichatclient-detection.md`: add the
  `AddOpenAIClient(...).AddChatClient(...)` row to the detection table,
  mark the Azure.AI.Inference row as legacy, and add a deprecation
  callout linking the Foundry migration guide.
- `references/common-pitfalls.md`: split the existing reasoning-model
  `max_tokens` pitfall into a short-term workaround (non-reasoning
  judge) and a durable fix (migrate to OpenAI/v1 SDK, which natively
  uses `max_completion_tokens`). Link the migration guide and
  dotnet/extensions#7580 which originally surfaced this for us.

No detection-logic changes: the legacy Azure.AI.Inference pattern is
still detected and supported exactly as before.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…EN phrasing

Two related doc cleanups across the three perf-skills:

1. Topology framing — .NET agentic apps are not always Aspire-based. The
   skills already worked against plain console / ASP.NET Core / worker
   service shapes, but the SKILL.md descriptions implied Aspire+Foundry
   were required. Broaden:
   - scan-agentic-app-perf: `(MAF + Aspire + Foundry)` -> `(MAF; Aspire/Foundry optional)`
   - setup-maf-evals:       `(MAF + Aspire + Foundry)` -> `(MAF; Aspire/Foundry optional)`
   - configure-agentic-perf-rules body: `typically with Aspire` -> `with or without Aspire/Foundry`
   Add a brief 'Supported topologies' body section to scan-agentic-app-perf
   and setup-maf-evals listing the four target shapes and noting that
   AppHost-specific checks are skipped when no AppHost is detected.

2. WHEN / NOT-WHEN phrasing harmonized across all three skills. The
   configure-agentic-perf-rules description was using
   `USE FOR` / `DO NOT USE FOR`; aligned with the other two on
   `WHEN` / `NOT-WHEN`.

Description char counts after trim stay under the 1024 cap:
- configure-agentic-perf-rules:  966
- scan-agentic-app-perf:         994
- setup-maf-evals:              1001

No detection-logic or scaffolding changes; pure doc.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

configure-agentic-perf-rules already had these sections; bring
scan-agentic-app-perf and setup-maf-evals up to the same shape per
CONTRIBUTING.md, which recommends 'When to Use' / 'When Not to Use' as
standard SKILL.md sections.

The frontmatter description still carries the high-signal WHEN /
NOT-WHEN entries for cheap LLM-side routing; the body sections expand
on each with rationale.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>